Unit III

Differences of Two Proportions

• Like with \(\hat{p}\), the difference of two sample proportions \(\hat{p}_1 - \hat{p}_2\) can be modeled using a normal distribution (when conditions are met).

\(SE = \sqrt{\dfrac{p_1 (1-p_1)}{n_1} + \dfrac{p_2 (1-p_2)}{n_2}}\)

• This standard error comes from the fact that variances of independent variables add, even when subtracting.

Differences of Two Proportions: Standard Errors

• When we talk about the spread of an estimate, we’re really talking about variance (the square of the standard error).

• If two random variables A and B are independent, then:

\(\text{Var}(A - B) = \text{Var}(A) + \text{Var}(B)\)

• This might seem counterintuitive — but remember:
  • Even if you’re subtracting two noisy measurements, the uncertainty (noise) from both still adds up.
  • Think of it like using two shaky rulers. Subtracting doesn’t cancel the shakiness — it just combines it!

Differences of Two Proportions: Simulation Setup

We’ll use simulation to understand the sampling distribution of sample proportions for two independent groups.

• Group 1 has a true proportion \(p_1 = 0.5\)
• Group 2 has a true proportion \(p_2 = 0.4\)
• Each group has \(n = 500\) individuals per sample
• We’ll repeat this sampling 1,000 times to observe variation in sample means

Differences of Two Proportions: Simulation

set.seed(1)       # ensures reproducibility
B <- 1000          # number of simulations
n <- 500           # sample size per group
p1 <- 0.5          # true proportion in group 1
p2 <- 0.4          # true proportion in group 2

# Create empty vectors to store simulated means and SDs
mean_x1 <- mean_x2 <- numeric(B)
sd_x1 <- sd_x2 <- numeric(B)

# Loop to simulate samples for both groups
for (i in 1:B) {
  # Generate random binary outcomes for group 1 (success = 1, failure = 0)
  x1 <- rbinom(n, size = 1, prob = p1)
  mean_x1[i] <- mean(x1)  # sample proportion for group 1
  sd_x1[i]   <- sd(x1)    # sample SD for group 1
  
  # Repeat for group 2
  x2 <- rbinom(n, size = 1, prob = p2)
  mean_x2[i] <- mean(x2)
  sd_x2[i]   <- sd(x2)
}

Comparing Theoretical and Empirical Standard Errors

• We can compare:
  • the theoretical standard error (from the formula)
  • the empirical standard error (from our simulations)

Comparing Theoretical and Empirical Standard Errors

# Theoretical standard error for a sample proportion
sqrt(p1*(1-p1)/n)

[1] 0.02236068

# Empirical standard error from simulated data
mean(sd_x1) / sqrt(n)

[1] 0.0223613

• The first line gives the theoretical SE: \(\sqrt{p(1-p)/n}\)
• The second line gives the empirical SE, based on simulated SDs
• These values should be nearly identical, validating the normal approximation for large \(n\)

Sampling Distribution for One Group (p1 = 0.5)

• We can visualize the distribution of sample proportions across simulations, and overlay a 95% confidence interval around the true mean.

Sampling Distribution for One Group (p1 = 0.5)

Sampling Distribution for One Group (p1 = 0.5)

• Roughly 5% of simulated sample proportions should fall outside this interval — confirming the 95% confidence level’s interpretation.

# Calculate the proportion of estimates that fall outside the 95% CI
mean(ifelse(mean_x1 >= ub_x1 | mean_x1 <= lb_x1, 1, 0))

[1] 0.052

Sampling Distribution for One Group (p2 = 0.4)

Sampling Distribution for One Group (p2 = 0.4)

• Again, around 5% of simulated estimates will fall outside the interval.
• The spread is slightly narrower than for Group 1 because the variance is smaller.

# Share of points outside the 95% CI
mean(ifelse(mean_x2 >= ub_x2 | mean_x2 <= lb_x2, 1, 0))

[1] 0.048

Combining Two Proportions

• Now that we understand the sampling variation of each group separately, we can combine them just as we would when estimating a difference in proportions:

\(SE_{\hat{p}_1 - \hat{p}_2} = \sqrt{SE_{\hat{p}_1}^2 + SE_{\hat{p}_2}^2}\)

• This formula reflects that variances add, even though we’re subtracting proportions.

Simulated Differences Between Groups

# Compute simulated differences
diff <- mean_x1 - mean_x2

# Theoretical SE for the difference in proportions
sqrt(p1*(1-p1)/n + p2*(1-p2)/n)

[1] 0.03130495

# Empirical SE estimate (not exact but illustrative)
mean(sd_x1 + sd_x2) / sqrt(n + n)

[1] 0.03129505

Simulated Differences Between Groups

Simulated Differences Between Groups

# Check coverage rates for both formulas
mean(ifelse(diff >= ub_diff_1 | diff <= lb_diff_1, 1, 0))

[1] 0.037

mean(ifelse(diff >= ub_diff_2 | diff <= lb_diff_2, 1, 0))

[1] 0.793

Differences of two proportions: Example 1

• Consider an experiment involving patients who underwent cardiopulmonary resuscitation (CPR) following a heart attack and were subsequently admitted to a hospital.
  • Patients were randomly assigned to either a treatment group (received a blood thinner) or a control group (no blood thinner).
  • The outcome of interest was survival for at least 24 hours.

Differences of two proportions: Example 1

Differences of two proportions: Example 1

• Create and interpret a 90% confidence interval of the difference for the survival rates in the CPR study.
  • \(p_t - p_c = 0.35 - 0.2 = 0.13\)
  • \(SE = \sqrt{\dfrac{0.35 (1-0.35)}{40} + \dfrac{0.22 (1-0.22)}{50}} \approx 0.095\)
  • \(0.13 \pm 1.645 \times 0.095 = (-0.027, 0.287)\)

Differences of two proportions: Computing in R

pt <- 14/40
pc <- 11/50
nt <- 40
nc <- 50

point_est <- pt - pc
se <- sqrt((pt * (1 - pt) / nt) + (pc * (1 - pc) / nc))

z <- data.frame(
  sig_level = c(0.01, 0.05, 0.1),
  z_score = c(2.45, 1.95, 1.645)
)

z$min <- point_est - z$z_score * se
z$max <- point_est + z$z_score * se
z

  sig_level z_score         min       max
1      0.01   2.450 -0.10396538 0.3639654
2      0.05   1.950 -0.05621734 0.3162173
3      0.10   1.645 -0.02709104 0.2870910

Differences of two proportions: Visualizing Confidence Intervals

Differences of two proportions: Interpretation

• We are 90% confident that blood thinners change the 24-hour survival rate by between -3 and 29 percentage points for patients similar to those in the study.

• Because 0% is within this range, the evidence is inconclusive — we cannot determine whether blood thinners help or harm heart attack patients who have undergone CPR.

Differences of Two Proportions: Example 2

• A 5-year clinical trial evaluated whether fish oil supplements reduce the risk of heart attacks.
  
• Each participant was randomly assigned to one of two groups:
  • Fish Oil group
  • Placebo group
• We’ll examine heart attack outcomes across both groups.

Differences of Two Proportions: Example 2

Differences of Two Proportions: Example 2

• Construct a 95% confidence interval for the effect of fish oil on heart attack incidence among patients represented by this study.
  
• Interpret the interval in context:
  • What does the direction and width of the interval suggest?
  • Is there evidence that fish oil has a meaningful effect on heart attack risk?

Differences of two proportions: Computing in R

nt <- 12933
nc <- 12938

pt <- 145 / nt
pc <- 200 / nc

point_est <- pt - pc
se <- sqrt((pt * (1 - pt) / nt) + (pc * (1 - pc) / nc))

z <- data.frame(
  sig_level = c(0.01, 0.05, 0.1),
  z_score = c(2.45, 1.95, 1.645)
)

z$min <- point_est - z$z_score * se
z$max <- point_est + z$z_score * se

Differences of two proportions: Visualizing Confidence Intervals

Differences of two proportions: Interpretation

• The point estimate for the effect of fish oil is approximately –0.0043,
  meaning heart attacks occurred 0.43 percentage points less often in the fish-oil group than in the placebo group.

• We are 90% confident that fish oil changes the heart-attack rate by between –0.66 and –0.19 percentage points for patients similar to those in the study.

• Because this interval does not include 0, the reduction in heart-attack risk is statistically significant at the 10% (and even 5% an 1%) level.

Practical vs. Statistical Significance

• While statistically significant, the effect size is extremely small — roughly 0.4 fewer heart attacks per 100 individuals.
  
• In a large clinical sample, even minor effects can reach significance if variability is low.
  
• From a practical standpoint, such a small reduction may not justify the cost, side effects, or adherence burden of treatment.

More on Two-Proportion Hypothesis Tests

• When conducting a two-proportion hypothesis test, the null hypothesis is typically: \(H_0: p_1 - p_2 = 0\)

• However, there are cases where we may want to test for a specific difference other than zero.

  • For example, suppose we want to test whether: \(H_0: p_1 - p_2 = 0.10\)

• In contexts like these, we use the sample proportions \[\hat{p}_1\] and \[\hat{p}_2\] to check the success–failure condition and to construct the standard error.

Differences of Two Proportions: Example 3

• A drone quadcopter company is considering a new manufacturer for rotor blades.

• The new manufacturer is more expensive but claims that their higher-quality blades are 3% more reliable, meaning that 3% more blades pass inspection compared to the current supplier.

• Set up the appropriate hypotheses for this test:

  • \(H_0: p_{\text{highQ}} - p_{\text{standard}} = 0.03\)
  • \(H_A: p_{\text{highQ}} - p_{\text{standard}} \neq 0.03\)

Differences of Two Proportions: Example 3 (Data)

• A quality control engineer collects samples of 1,000 blades from each manufacturer:
  • Current supplier: 899 blades pass inspection
    
  • Prospective supplier: 958 blades pass inspection
• Using these data, evaluate the hypotheses above at a significance level of 5%.

Compute the Point Estimate and Standard Error

p_us <- 958 / 1000
p_them <- 899 / 1000
point_est <- p_us - p_them

# Standard error for independent samples
se <- sqrt( p_us * (1 - p_us) / 1000 +  p_them * (1 - p_them) / 1000 )

p_us; p_them

[1] 0.958

[1] 0.899

point_est

[1] 0.059

se

[1] 0.01144705

Compute Confidence Intervals for Various Significance Levels

z <- data.frame(
  sig_level = c(0.01, 0.05, 0.10),
  z_score = c(2.45, 1.95, 1.645)
)

z$min <- (point_est - z$z_score * se) - 0.03
z$max <- (point_est + z$z_score * se) - 0.03
z

  sig_level z_score          min        max
1      0.01   2.450 0.0009547225 0.05704528
2      0.05   1.950 0.0066782486 0.05132175
3      0.10   1.645 0.0101695994 0.04783040

Visualizing Confidence Intervals

Compute and Visualize the z-Statistic

z <- (point_est - 0.03) / se

set.seed(1)
sim <- rnorm(1000, mean = 0.03, sd = se)

# Probability of observing a value this extreme or larger
1 - mean(ifelse(point_est >= sim, 1, 0))

[1] 0.004

# p-value (right-tailed)
1 - pnorm(z)

[1] 0.005648044

Visualizing the Sampling Distribution

Example 3: Conclusion

• From the standard normal distribution:
  • The right-tail area is approximately 0.004
  • Doubling for a two-tailed test gives p = 0.008
  • Since p = 0.008 < 0.05, we reject the null hypothesis
• We find statistically significant evidence that the higher-quality blades have a pass rate greater than 3% higher than the standard blades — exceeding the company’s claims.

Chi-Squared Distributions: Introduction

• \(\chi\) = the greek letter for “chi” (pronounced like “kai”)

• The \(\chi^2\) distribution is a continous probability distribution that is widely used in statistical inference.

  • Closely related to the standard normal distribution

• If a variable \(Z\) has the standard normal distribution, then \(Z^2\) has the \(\chi^2\) distribution with one degree of freedom

Chi-Squared Distributions: Histograms

Chi-Squared Distributions: Definition

• If \(Z_1, Z_2,..., Z_k\) are independent standard normal variables, then…
  • \(Z_1^2 + Z_2^2 +...+ Z_k^2\)
  • …has a \(\chi^2\) distribution with \(k\) degrees of freedom.

Degrees of Freedom: Concept

• A degree of freedom (df) represents the number of independent pieces of information available to estimate something.

• Whenever we calculate a statistic, we “use up” some information.

  • For example, once we estimate the sample mean, one data point can be perfectly predicted from the others.
  • So for a sample of size \(n\), only \((n - 1)\) observations are free to vary when computing the sample variance.

Degrees of Freedom: Intuition

• Average (mean): \(\bar{x} = \frac{1}{n} \sum_i^n x_i = \frac{x_1 + ... + x_n}{n}\)

• if \(i = 4\), \(x_1 = 8\), \(x_2 = 10\), \(x_3 = 12\) and \(\bar{x} = 10\)…

  • … then \(x_4 = 10\)

• So even though we had four data points, only three were free to vary — the fourth is determined by the mean.

• That’s why when calculating the sample variance, we divide by \(n\) instead of \(n-1\): one degree of freedom has been “used up” in estimating the mean.

Degrees of Freedom: Motivation

• Why it matters:
  • Degrees of freedom tell us how much independent information our test or estimate is based on.
  • They affect the shape of sampling distributions (like \(t\), \(F\), and \(\chi^2\)), which in turn changes the critical values and p-values we use.
  • More degrees of freedom → more information → the distribution becomes narrower and more normal-looking.
• Big idea:
  • Degrees of freedom link sample size, uncertainty, and the reliability of inference — they remind us that every time we estimate something, we “spend” information.

Chi-Squared Distributions: Properties

• mean: \(\mu = k\)
• variance: \(\sigma^2 = 2k\)
• mode occurs at \(\mu - 2\)

set.seed(1)
x <- rnorm(1000, mean = 0, sd = 1)
x2 <- x^2
mean(x2)

[1] 1.070115

var(x2)

[1] 2.291541

Chi-Squared Distributions: Multiple Degrees of Freedom

Chi-Squared Distributions: Multiple Degrees of Freedom

n_loops <- 2
z <- list()
for(i in 1:n_loops){
  set.seed(i)
  x <- rnorm(1000, mean = 0, sd = 1)
  x2 <- x^2
  if(i == 1){
    z2 <- x2
  }else{
    z2 <- z2 + x2
  }
}
mean(z2); n_loops

[1] 2.103046

[1] 2

var(z2); 2*n_loops

[1] 4.281053

[1] 4

Chi-Squared Distributions: Multiple Degrees of Freedom

n_loops <- 20
z <- list()
for(i in 1:n_loops){
  set.seed(i)
  x <- rnorm(1000, mean = 0, sd = 1)
  x2 <- x^2
  if(i == 1){
    z2 <- x2
  }else{
    z2 <- z2 + x2
  }
}
mean(z2); n_loops

[1] 20.08944

[1] 20

var(z2); 2*n_loops

[1] 40.06536

[1] 40

Chi-Squared Distributions: Multiple Degrees of Freedom